Human Reasoning and Cognitive Science

subjunctive 13 2 8 3* 0 1 2 1 1 0 0 31 TABLE 4. Frequencies of card choice combinations by conditions. The modified conjunction task and its new baseline condition are below the earlier results which are repeated here for convenience. Classical logical competence responses are marked *. Any response made by at least three subjects in at least one condition is categorised: everything else is miscellaneous The results on the conjunctive rule illustrate several general issues: how easy it is to invoke a deontic reading of indicative wording; how unnatural it is for naive subjects to adopt an ‘is-this-sentence-literally-true’ perspective rather than a ‘what-are-the-experimenter’s-intentions’ perspective; that the difficulty of classical interpretation can be as great with conjunction as with implication. Although the difficulties may be different difficulties, there is a real possibility that they are closely related through conjunctive suppositional interpretations of the conditional. Finally, we explored one other obvious manipulation designed to follow up the malleability of subjects’ interpretations exposed by the conjunctive rule. This is worth discussing as a ‘failed’ negative result as much as for its own interest. If subjects’ difficulties in the original descriptive task follow from the complexities of descriptive semantics, is it possible to restore deontic levels of performance in the abstract task merely by simply making the rule subjunctive? We ran a further condition in which the rule used was: 3. DISCUSSION OF RESULTS 109 If a card has a vowel on one side, then it should have an even number on the other. and the instruction was to choose which of the four cards you must turn in order to decide if the card complies with the rule. The results of this condition are shown in Table 4 in the ‘Abstract subjunctive’ row. Three subjects of 31 turned P and not-Q, as compared to one of 30 in the baseline. If this is a facilitation it is a small one (the probability of getting this facilitation by chance is 0.32 i.e. statistically insignificant). Merely using subjunctive wording may be insufficient to invoke a deontic reading. This is not so surprising since there is an alternative ‘epistemic’ interpretation of the subjunctive modal here which might still be used with a descriptive semantics for the underlying rule. Imagine that the rule is clearly a robust descriptive scientific law (perhaps ‘All ravens are black’), then one might easily state in this context, that a card with ‘raven’ on one side should have ‘black’ on the other, implying something about what the cards have to be like to comply with the scientific law (still with a descriptive semantics underlying), rather than what the birds have to do to comply with a legal regulation. This possibility of interpretation may make it hard to invoke a deontic interpretation without further contentful support. Contentful support is, of course, what the various ‘quality inspector’ scenarios provide (e.g. [204]). Contentful support is also what permission and obligation schemas, and the ‘seek violations’ instructions in combination with modal explications of the rule provide, as reported by Platt and Griggs [173]. This failure of what is about the simplest direct manipulation that could test the theory (that it is deontic interpretation which achieves most of the facilitations reported in the literature) is a very common kind of failure. Our interpretation is that the manipulation unfortunately fails to achieve its goal. An opponent of our theory might claim it as evidence the theory is wrong. To settle this dispute one would need to get further evidence that the manipulation doesn’t succeed in invoking a deontic reading. Trying such direct tests is important – if they work they provide dramatic evidence. Generally when they fail they are not reported. The problem is a problem of bridging from a theory of interpretation to the effects of manipulations, and bridging is unavoidable if theory is to be brought to bear on data. In summary of all the conditions, these results corroborate the findings of the tutoring experiments, also reported in [215], that our manipulations alleviate real sources of difficulty with interpretation for subjects in the original descriptive task – sources of difficulty which do not apply in the deontic task. This evidence suggests that far from failing to think at all, subjects are sensitive to several important semantic issues posed by the descriptive task. Since our manipulations only help to alleviate problems piecemeal in some subjects, they cannot be expected to install a classical logical interpretation of the task and materials in all subjects, nor to produce performance levels comparable with the deontic tasks. It is difficult to assess what proportion 110 4. FROM LOGIC, VIA EXPLORATION, TO CONTROLLED EXPERIMENT of problems have been dealt with here. Several that we discussed in chapter 3 have not been treated (anaphora, card reversibility, . . . ). It is an even harder methodological problem to find a way of estimating how much of the difficulties these problems jointly account for, because simply applying all the manipulations concurrently produces complicated instructions which themselves introduce their own problems. Rather than presenting a complete model of what subjects are doing in this task, the experiment provides strong evidence that several of the major sources of interpretation problems identified by semantic analysis do contribute to subjects’ difficulties in the original task. This is sufficient to make it very hard to defend any theory which assumes both that the correct interpretation is classical logic, and is the interpretation that subjects adopt. This includes all theories except the information-gain theory. As discussed above, both the two-rule and contingency conditions provide considerable evidence that the information-gain theory has its own problems. An ingenious experiment by [?] is aimed at the difficult task of assessing how many of these problems can be simultaneously alleviated. How many subjects can be induced to adopt the material implication interpretation Wason’s criterion of correct reasoning requires, by simply simultaneously removing problems Wason strews in the subjects’ path? And how many subjects then reason as theit chosen interpretation requires? [173] induced very high selections of A and 7 cards in the descriptive task by focusing subjects on possibly non-compliant cases. But van Denderen’s aim was to see what could be achieved without this focusing. The manipulations he used are as follows: 1) the task was run interactively on the web so that subjects could actually turn cards and alleviate the problems of wanting to make responses contingent on the feedback of early turns; 2) the conditional is not described as a rule (with the result of engendering a defeasible interpretation of the relevance of a larger population of cards) but rather as a simple expression of an observer’s belief about four cards; 3) the conditionality (as opposed to biconditionality) of the rule is emphasised by inclusion of a clause explicitly disavowing the biconditional reading; 4) the source of the conditional was separated from the experimenter to avoid issues of the need to accuse an authority figure of lying; and 5) the subject is required to confirm their understanding of the instructions by answering some simple questions about the task before proceeding to the experiment. The design of the experiment divided subjects into a group with a simple conditional (original conditional), and a group with the clause rejecting the biconditional added (extended rule). All subjects did two versions of the task: one including all the enhancements just listed except for the possibility of interacting with the cards (non-interactive), and the other with this interaction (interactive) , in that order. After these two selection tasks, subjects did an ‘evaluation task’ [243] where they judged each of the four card types as complying, not complying, or irrelevant to the conditional, in order to gather data on their interpretation. Armed with this data about 4. WHAT DOES THIS SAY ABOUT REASONING MORE GENERALLY? 111 individual interpretation, it is possible to assess not only whether subjects adopt Wason’s intended interpretation of the conditional, but also whether they reason correctly from it, or from whatever other interpretation they do adopt. The results show a huge convergence on Wason’s desired interpretation of the conditionals. Whereas with the original conditional, only a fifth of subjects chose the material implication interpretation (even with all the clarifications), with the extended rule this rose to 80%. There was a similarly large increase of A and 7 selections. From around 5% making this choice in Wason’s original experiment, 32% of subjects in the extended rule interactive task chose just A and 7 and a further 16% chose A, 7 and 4. All told 60% turned the 7 card compared to about 9% in Wason’s original task. About 55% of subjects made all the the card choices in the complex interactive task which normatively corresponded with their chosen interpretation, although more than half of these chose one or more cards they strictly need not have turned. These results are a strong vindication of the theory we have proposed here. There is much interpretative variability, but a large majority of subjects can be induced to adopt the interpretation Wason intended, and when they do, a large proportion of them reason adequately from that interpretation. Reasoning is by no means perfect, but the hard part is getting to the interpretation which Wason hid. This experiment is rather revealing of the forest of problems the subject has to navigate, and therefore suggestive of why the ability to find Wason’s interpretation and turn the A and 7 cards should be predictive of academic success as Stanovich showed. 4. What does this say about reasoning more generally? What implications do these results have for theories of reasoning, and for the place of interpretation in cognitive theory more generally? What do they tell us about the way the field has viewed the relation between logical and psychological analyses of reasoning, and how that relation might be construed more productively? Each theory is a somewhat different case. These results remove the founding evidence for ‘evolutionary’ theories which propose that the difference in performance on ‘social contract’ conditionals and descriptive conditionals needs to be explained by innate cheating detection modules evolved in the Pleistocene. This topic will be discussed at greater length in Chapter 5. More generally, this reappraisal of the selection task provides a good example of how arguments for ‘massive modularity’ in cognition should be treated with some scepticism (for further arguments see [218]). The original experiments found variation in performance as a function of difference in materials. Sweeping generalisations were then made from the laboratory task without any consideration of the relation between that task and subjects’ other communication and reasoning abilities. Just as our analysis directs attention to the differences between variations on the selection 112 4. FROM LOGIC, VIA EXPLORATION, TO CONTROLLED EXPERIMENT task and the continuities between natural language communication inside and outside the selection task, so our proposals return attention to the issue how humans’ generalised communication capacities arose in evolution. The interactions between logic’s dual apparatus of interpretation and of derivation constitute an exquisitely context sensitive conceptual framework for the study of human reasoning and communication, whether in evolution, development or education. Chapter 5 takes up these themes. The non-evolutionary theories of human reasoning are most generally affected by the present results through their implications for the relation between logic and psychology. We focus here particularly on relevance theory, mental models theory, and rational analysis models. Inasmuch as relevance theory assumes that human reasoning and communication abilities are general abilities which interact with the specifics of the context, our general drift is sympathetic to relevance theory’s conclusions. We agree with relevance theory that the goal must be to make sense of what subjects are doing in the very strange situation of laboratory reasoning tasks – in a memorable phrase, to see subjects as ‘pragmatic virtuosos’ (Girotto et al. [78]) – rather than to see them as logical defectives. Our divergences from relevance theory are about the granularity of interaction between semantic and pragmatic processes in subjects’ reasoning; in the range of behaviour we believe to be of theoretical concern; and in the program of research. Relevance Theory explains pragmatic effects in terms of very general factors – relevance to the task at hand and cost of inference to reveal that relevance. These factors must always operate with regard to some semantic characterisation of the language processed. Condensing analysis into these two pragmatic factors however seems, in this case at least, to have led to relevance theorists missing the critical semantic differences which drive the psychological processes in this task – the differences between deontic and descriptives and their consequences for interpretation in this task’s setting. Relevance theory’s conclusion has been that not much reasoning goes on when undergraduate subjects get the abstract task ‘wrong’. Our combination of tutoring observations and experiment strongly suggest that a great deal goes on, however speedily the ‘pre-computed’ attitudes are brought to bear in the actual task, and that the exact nature of the processes is highly variable from subject to subject. Taking logic more seriously leads us to seek more detailed accounts of mental processes. The current results have rather wide-ranging implications for mental models theory. Some implications specific to the theory’s application to the selection task have already been discussed. Others are more general, about mental models theory’s relation to logic and semantics. Since JohnsonLaird’s early work with Wason on the selection task mental models theory has been elaborated by a complex theory of the meanings of conditionals and the overlay of semantics by ‘pragmatic modulation’, and the theory has been much exercised by the issue whether subjects’ interpretations of the 4. WHAT DOES THIS SAY ABOUT REASONING MORE GENERALLY? 113 rule in the selection task is truth-functional or not [115]. However, this consideration of semantic possibilities has been divorced from any consideration of their implications for the subjects’ interpretation of the task. If subjects’ reading of the rule is non-truth-functional (by whatever semantic or pragmatic route), then the subject should experience a conflict between their interpretation and the task instructions. This conflict has never been acknowledged by mental models theorists. What justification can there then be for applying the classical logical competence model as a criterion of correct performance while simultaneously rejecting it as an account of how subjects interpret the conditional? We shall return to some of the general logical issues raised by mental models in Chapter 5, section 1.4 and Chapter 9, section 5. Finally, where do our findings leave the rational analysis models of selection task behaviour as optimal experiment (Oaksford and Chater [163]). We applaud these authors’ challenge to the uniqueness of the classical logical model of the task, and also their insistence that the deontic and descriptive versions of the task require distinct accounts. Their theory is clearly more sophisticated about the relations between formal models and cognitive processes than the theories it challenges. However, our proposals are quite divergent in their cognitive consequences. The rational analysis models reject any role for logic, claiming that the task is an inductive one. But this move smuggles logic in the back door. Applying optimal experiment theory requires assigning probabilities to propositions, and propositions are specified in some underlying language. The logic underlying the rational analysis model is the same old classical propositional calculus with all its attendant divergences from subjects’ interpretations of the task materials. This has direct psychological consequences. The rational analysis models treat subjects’ performances as being equally correct as measured by the two distinct competence models for descriptive and deontic tasks. Our analysis predicts that the descriptive task will be highly problematical and the deontic task rather straightforward. The tutorial evidence on the descriptive task and its experimental corroboration support our prediction about the descriptive task. Approaching through interpretation predicts and observes considerable variety in the problems different subjects exhibit in the descriptive task, and even variety within the same subject at different times. We can agree that some subjects may adopt something like the rational analysis model of the task, but disagree about the uniformity of this or any other interpretation. Most of all we do not accept that everyone is doing the same thing at the relevant level of detail. This situates our approach with regard to some prominent psychological theories of reasoning, and illustrates similarities and differences with extant approaches in the context of this one particular task. But our proposals also have general implications for how cognitive theories of reasoning relate to logical and linguistic theories of language and communication more generally. If we are anything like right about the selection task, it is 114 4. FROM LOGIC, VIA EXPLORATION, TO CONTROLLED EXPERIMENT both possible and necessary to bring the details of formal accounts of natural languages (semantics of deontics and descriptives, variable and constant anaphora, tense, definiteness, domain of interpretation, scope of negation, . . . ) to bear in explaining the details of performance in laboratory reasoning tasks. This is necessary because subjects’ behaviour in these tasks is continuous with generalised human capacities for communication, and possible because although strange in many ways, laboratory tasks have to be construed by subjects using their customary communicative skills. In fact laboratory tasks have much in common with the curious communicative situation that is formal education, and another benefit of the current approach is that it stands to reconnect the psychology of reasoning with educational investigations. With very few exceptions (e.g. Stanovich and West [205]), psychologists of reasoning have not asked what educational significance their results have. They regard their theories as investigating ‘the fundamental human reasoning mechanism’ which is independent of education. On our account, the descriptive selection task is interesting precisely because it forces subjects to reason in vacuo and this process is closely related to extremely salient educational processes which are aimed exactly at equipping students with generalisable skills for reasoning in novel contexts more effectively. For example, the balance of required co-operative assumption of the background rule and adversarial test of the foreground rule in the descriptive selection task, is absolutely typical of the difficulties posed in the strange communications involved in examination questions (cf. the quote from Ryle on page 2). Many cross-cultural observations of reasoning can be understood in terms of the kinds of discourse different cultures invoke in various circumstances (see Chapter 5). The discourses established by formal education are indeed a very distinctive characteristic of our culture (see e.g. [14]; [99]).